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' Abstract 

o ■ 

I We introduce S-DUDE, a new algorithm for denoising DMC-corrupted data. The algorithm, which 

. generalizes the recently introduced DUDE (Discrete Universal DEnoiser) of Weissman et al., aims to 

bJ)| compete with a genie that has access, in addition to the noisy data, also to the underlying clean data, 

^ I and can choose to switch, up to m times, between sliding window denoisers in a way that minimizes 

I the overall loss. When the underlying data form an individual sequence, we show that the S-DUDE 

. performs essentially as well as this genie, provided that m is sub-linear in the size of the data. When the 

' clean data is emitted by a piecewise stationary process, we show that the S-DUDE achieves the optimum 
distribution-dependent performance, provided that the same sub-linearity condition is imposed on the 

r . I number of switches. To further substantiate the universal optimality of the S-DUDE, we show that when 

. the number of switches is allowed to grow linearly with the size of the data, any (sequence of) scheme(s) 

^ ' fails to compete in the above senses. Using dynamic programming, we derive an efhcient implementation 

O . of the S-DUDE, which has complexity (time and memory) growing only linearly with the data size and 
the number of switches m. Preliminary experimental results are presented, suggesting that S-DUDE has 

y—i . the capacity to significantly improve on the performance attained by the original DUDE in applications 

^ ' where the nature of the data abruptly changes in time (or space), as is often the case in practice. 

^ ! 
\o . 

, piecewise stationary processes, dynamic programming, discrete memoryless channel (DMC), switching 

• ' experts, forward-backward recursions. 

00 ■ 

o : 

^ ; 1 Introduction 

. I— I ; Discrete denoising is the problem of reconstructing the components of a finite-alphabet sequence based on 
^ ■ the entire observation of its Discrete Memoryless Channel (DMC)-corrupted version. The quahty of the 
' reconstruction is evaluated via a user-specified (single-letter) loss function. Universal discrete denoising, 
in which no statistical or other properties are known a priori about the underlying clean data and the goal 
is to attain optimum performance, was considered and solved in [1]. The main problem setting there is 
the "semi-stochastic" one, in which the underlying signal is assumed to be an "individual sequence," and 
the randomness is due solely to the channel noise. In this setting, it is unreasonable to expect to attain 
the best performance among all the denoisers in the world, since for every given sequence, there exists a 
denoiser that recovers all the sequence components perfectly. Thus, [1] limits the comparison class, a.k.a. 
expert class, and uses the competitive analysis approach. Specifically, it is shown that regardless of what 
the underlying individual sequence may be, the Discrete Universal DEnoiser (DUDE) essentially attains 
the performance of the best sliding window denoiser that would be chosen by a genie with access to the 
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underlying clean sequence, in addition to the observed noisy sequence. This semi-stochastic setting result 
is shown in [1] to imply the stochastic setting result, i.e., that for any underlying stationary signal, the 
DUDE attains the optimal distribution-dependent performance. 

The setting of an arbitrary individual sequence, combined with competitive analysis, has been very 
popular in many other research areas, especially for problems of sequential decision-making. Examples 
include universal compression [4], universal prediction [5], universal filtering [2], repeated game playing 
[6, 7, 8], universal portfolios [9], online learning [10, 11], zero-delay coding [12, 13], and much more. A 
comprehensive account of this line of research can be found in [14]. The beauty of this approach is the 
fact that it leads to the construction of schemes that perform, on every individual sequence, essentially as 
well as the best in a class of experts, which is the performance of a genie that had hindsight on the entire 
sequence before selecting his actions. Moreover, if the expert class is judiciously chosen, the relative sense 
of such a performance guarantees can, in many cases, imply optimum performance in absolute senses as 
well. 

One extension to this approach is competition with an expert class and a genie that has the freedom to 
form a compound action, which breaks the sequence into a certain (limited) number of segments, applies 
different experts in each segment, and achieves an even better performance overall. Note that the optimal 
segmentation of the sequence and the choice of the best expert in each segment is also determined by hind- 
sight. Clearly, competing with the best compound action is more challenging, since the number of possible 
compound actions is exponential in the sequence length n, and the brute-force vanilla implementation of 
the ordinary universal scheme requires prohibitive complexity. However, clever schemes with linear com- 
plexity that successfully track the best segments and experts have been devised in many different areas, 
such as online learning, universal prediction [15, 16], universal compression [17, 18], online linear regression 
[19], universal portfolios [20], and zero-delay lossy source coding [22]. 

In this paper, we expand the idea of compound actions and apply it to the discrete denoising problem. 
The motivation of this expansion is natural: the characteristics of the underlying data in the denoising 
problem often tend to be time- or space-varying. In this case, determining the best segmentation and the 
best expert for each segment requires complete knowledge of both clean and noisy sequences. Therefore, 
whereas the challenge in sequential decision-making problems is to track the shift of the best expert based 
on the past, true observation, the challenge in the denoising problem is to learn the shift based on the 
entire, but noisy, observation. We extend DUDE to meet this challenge and provide results that parallel 
and strengthen those of [1] . 

Specifically, we introduce the S-DUDE and show first that, for every underlying noiseless sequence, 
it attains the performance of the best compound finite-order sliding window denoiser (concretely defined 
later), both in expectation and in a high probability sense. We develop our scheme in the semi-stochastic 
setting as in [1]. The toolbox for the construction and analysis of our scheme draws on ideas developed 
in [2]. We circumvent the difficulty of not knowing the exact true loss by using an observable unbiased 
estimate of it. This kind of an estimate has proved to be very useful in [2] and [3] to devise schemes 
for filtering and for denoising with dynamic contexts. Building on this semi-stochastic setting result, we 
also establish a stochastic setting result, which can be thought of as a generalization and strengthening of 
the stochastic setting results of [1], from the world of stationary processes to that of piecewise stationary 
processes. 

Our stochastic setting has connections to other areas, such as change-point detection problems in 
statistics [23, 24] and switching linear dynamical systems in machine learning and signal processing [25, 26]. 
Both of these lines of research share a common approach with S-DUDE, in that they try to learn the change 
of the underlying time-varying parameter or state of stochastic models, based on noisy observations of the 
parameter or state. One difference is that, whereas our goal is the noncausal estimation, i.e., denoising, 
of the general underlying piecewise stationary process, the change-point detection problems mainly focus 
on sequentially detecting the time point where the change of model happened. Another difference is in 
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that the switching hnear dynamical systems focus on a special class of underlying processes, the linear 
dynamical system. In addition, they deal with continuous-valued signals, whereas our focus is the discrete 
case, with finite-alphabet signals. 

As we explain in detail, the S-DUDE can be practically implemented using a two-pass algorithm with 
complexity (both space and time) linear in the sequence length and the number of switches. We also 
present initial experimental results that demonstrate the S-DUDE's potential to outperform the DUDE on 
both simulated and real data. 

The remainder of the paper is organized as follows. Section 2 provides the notation, preliminaries and 
background for the paper; in Section 3 wc present our scheme and establish its strong universality properties 
via an analysis of its performance in the semi-stochastic setting. Section 4 establishes the universality of 
our scheme in a fully stochastic setting, where the underlying noiseless sequence is emitted by a piecewise 
stationary process. Algorithmic aspects and complexity of the actual implementation of the scheme is 
considered in Section 5, and some experimental results are displayed in Section 6. In Section 7 we conclude 
with a summary of our findings and some possible future research directions. 

2 Notation, Preliminaries, and Motivation 
2-A Notation 

We use a combination of notation of [1] and [2]. Let X, Z, X denote, respectively, the alphabet of the clean, 
noisy, and reconstructed sources, which are assumed to be finite. As in [1] and [2], the noisy sequence is a 
DMC-corrupted version of the clean one, where the channel matrix 11 = {n(x, z)}xex,zez-, n(x, z) denoting 
the probability of a noisy symbol z when the underlying clean symbol is x, is assumed to be known and 
fixed throughout the paper, and of full row rank. The z-i\i column of 11 will be denoted as tt^. Upper case 
letters will denote random variables as usual; lower case letters will denote either individual deterministic 
quantities or specific realizations of random variables. 

Without loss of generality, the elements of any finite set V will be identified with {0, 1, • • • , |V| — 1}. We 
let V°° denote the set of one-sided infinite sequences with V- valued components, i.e., v G V°° is of the form 
V = (wi, f2, ■ ■ ■)-,Vi ,i > 1. For V G V°°, let = (ui, • • • , f„) and = [vm-, • • • , Vn)- Furthermore, we 
let denote the sequence v^~^v^_^i. is a space of |V|-dimensional column vectors with real-valued 
components indexed by the elements of V. The a-th component of g G will be denoted by either qa 
or q[a]. Subscripting a vector or a matrix by "max" will represent the difference between the maximum 
and minimum of all its components. Thus, for example, if F is a \Z\ x \X\ matrix, then Fmax stands 
for maxx£x,zez^iz,x) — miiix^x ^zez ^ (z, x) (in particular, if the components of F are nonnegative and 
T{z,x) = for some z and x, then F^ax = max^g^ ^g;^' F(z, x).) In addition, denotes an indicator of 
the event inside {•}. 

Generally, let the finite sets y, A be, respectively, a source alphabet and an action space. For a general 
loss function I : y x A a Bayes response for ^ G M.-^ under the loss function I is given as 



where denotes the column of the matrix of the loss function I corresponding to the a-th action, and ties 
are resolved lexicographically. The corresponding Bayes envelope is denoted as 



Note that when (" is a probability, namely, it has non-negative components summing to one, Ui{C) is the 
minimum achievable expected loss (as measured under the loss function I) in guessing the value ofY€y 
which is distributed according to (. The associated optimal guess is 6/(C)- 



HO = argminC^ • C, 



(1) 



i7KC)=minC^-£„. 



(2) 
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An n-block denoiser is a collection of n mappings X"^ = {Xt}i<t<m where Xt : Z"' X. We assume 
a given loss function A : A" x A* ^ [0, oo), where the maximum single-letter loss is denoted by Amax, and 
A-r denotes the x-th column of the loss matrix. The normalized cumulative loss of the denoiser X" on the 
individual sequence pair (x^jZ") is represented as 

L^„(x",0 = i^A(xt,Xt(z")). 

' t=i 

In words, L-j^„ (z", z") is the normalized (pcr-symbol) loss, as measured under the loss function A, when 
using the denoiser X" and when the observed noisy sequence is while the underlying clean one is x". 
The notation L-^„ is extended for 1 < z < j < n, 

^ t=i 

denoting the normalized (per-symbol) loss between (and including) locations i and j. 

Now, consider the set S = {s : Z ^ X}, which is the (finite) set of mappings that take Z into X. 
We refer to elements of 5 as "single-symbol denoisers" , since each s G 5 can be thought of as a rule for 
estimating X G X on the basis oi Z & Z. Now, for any s G <S, an unbiased estimator for A(x, s{Z)) (based 
on Z only), where x is a deterministic symbol and Z is the output of the DMC when the input is x, can 
be obtained as in [2]. First, pick a function h : Z ^ M'^ with the property that, for a,b G X, 

EahiZ) = J2^b{z)U{a,z) 

= "^^"'^^-j o! ofherwise }' 

where Ea denotes expectation over the channel output Z when the underlying channel input is a, and 
hb{z) denotes the 6-th component of h{z). Let H denote the \Z\ x \X\ matrix whose z-ih. row is h'^{z), 
i.e., H{z,b) = hi,{z). To sec that our assumption of a channel matrix with full row rank guarantees the 
existence of such an h, note that (3) can equivalently be stated in matrix form as 

UH = I, (4) 

where / is the \X\ x \X\ identity matrix. Thus, e.g., any H of the form H = r-^(nr^)~^, for any F such 
that ITF"^ is invertible, satisfies (4). In particular, F = 11 is a valid choice (IIII^ is invertible, since 11 is of 
full row rank) corresponding to the Moore-Penrose generalized inverse [27]. Now, for any s G 5, p{s) G M'^ 
denotes the column vector with x-th component 

Px{s) = ^A{x,s{z))U{x,z) 

z 

= E^A{x,s{Z)). (5) 

In words, Px{s) is the expected loss using the single-symbol denoiser s, while the underlying symbol is x. 
Considering <S as an action space alphabet, we define a loss function i : Z x S ^ M. as 

e{z,s) = h{zf-p{s). (6) 

We observe from (3) and (5) that i{Z,s) is an unbiased estimate of A(x,,s(Z)) since 

EJ{Z,s) = E^hiZf ■ p{s) = Y,ExK'{Z)px'{s) = ^5(x,x')p,.'(s) = Px{s) = ExA{x,s{Z)) Vx G X. (7) 

x' x' 
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For ^ G M^, let Bni^, ■) & S he defined by 

Bh {^,z) = arg min • ■ [A^ tt^] , (8) 

X 

where, for vectors vi and V2 of equal dimensions, vi V2 denotes the vector obtained by component- wise 
multiplication. Note that, similarly as in [2, (88), (89) ], 

BH{i,-) = argmin^e^.if.[A,(,)0 7r,] 

z 

= arg min ■ H ■ p{s) 

s&S 

= arg min V • [h^ {z) ■ p{s)\ 

s&o — 

z 

= arg min •^(2:,s) = 5^(0- (9) 

z 

Thus, Bh{^, ■) is a Bayes response for ^ under the loss function £ defined in (6). 
2-B PreliminEiries 

In this section, we summarize the results from [1] and motivate the approach underlying the construction of 
our new class of denoisers. Analogously as in [2], the ra-block denoiser X" = {Xt}i<t<n can be associated 
with F" = {Ft}i<t<n, where Ft : 2^"^* S is defined as follows: Ft{z^\^, •) is the single-symbol denoiser 
in S satisfying 

Xt(z")=Fi(z"\*,Zi) \/zt. (10) 

Therefore, we can adopt the view that at each time t, an n-block denoiser is choosing a single-symbol 
denoiser based on all the noisy sequence components but zt, and applying that single-symbol denoiser on 
zt to yield the t-th reconstruction xt- Conversely, any sequence of mappings into single-symbol denoisers 
F" defines a denoiser X", again via (10). We will adhere to this viewpoint in what follows. 

One special class of widely used n-block denoisers is that of k-th order "sliding window" denoisers, 
which we denote by X"''^'= . Such denoisers are of the form 

X^'{z^) = Sk{zlt^^), t = k + l,--- ,n-k, (11) 

where Sk is an element of Sk = {sk ■ Z'^^'^^ X}, the (finite) set of mappings from Z^'^+i into X} We 
also refer to Sk G (Sfc as a "A;-th order denoiser". Note that Sq = S. Prom the definition (11), it follows that 

je;'=(z") =X;'=(z") whenever = (12) 

Following the association in (10), we can adopt an alternative view that the k-th order sliding window 
denoiser chooses a single-symbol denoiser Skiz^zl^ z^^i, •) € <S at time t on the basis of the context, and 
Xt^z-) = Sk{zlzlzlXlzt). 

We denote Ct = {zlzl z^+'l) as a (two-sided) context for zt, and define the set of all possible A;-th order 
contexts, = {{uZl.,Ui) : ('ul^,u^) G Z^*^}. Then, for given and for each c G Cj^, we define 

T{c) = {t:ct = c, k + l<t<n-k} = {t:{zlzl,zlX'l) = c, k + l<t<n-k}, (13) 
^The value of X^'' («") ior t < k and t > n — k is defined, for concreteness and simplicity, as an arbitrary fixed symbol in 

X. 
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the set of indices where the context equals c. Now, an equivalent interpretation for (12) is that for each 
c G Cfc, the A;-th order sliding window denoiser employs a time-invariant single-symbol denoiser, Sfe(c, •), 
at all points t G T(c). In other words, the sequence 2" is partitioned into the subsequences associated with 
the various contexts, and on each such subsequence a time-invariant single-symbol scheme is employed. 
In [1], for integers A; > and n > 2k, the A;-th order minimum loss of (x", z") is defined by 

L>fe(x",^") ^ min Lx„(47f,^") 



n—k 

skeSk n — 2k 



mill — — A{xt,sk{ct,zt)). (14) 



t=k+l 



The identity of the element Sk G Sj- that achieves (14) depends not only on z", but also on x", since (14) 
can be expressed as 

min ^ A{xr,s{zr)) 



n-2k ^ 

ceCk 



rer(c) 



and at each time t, the best A;-th order sliding window denoiser that achieves (14) will employ the single- 
symbol denoiser 

argmin A{xr,s{zr)), (15) 

rer(ct) 

which is determined from the joint empirical distribution of pairs {{xr, z^) : t € T{ct)}. 

It was shown in [1] that, despite the lack of knowledge of x", Dk{x^, Z") is achievable in a sense made 
precise below, in the limit of growing n, by a scheme that only has access to Z". This scheme is dubbed 
in [1] as the Discrete Universal DEnoiser (DUDE), X^^^^. The algorithm is defined by 

^u'niv,t(^") = BH{m{z-,zlzl4X'[),zt), (16) 

where m(z'*, c) is the vector of counts of the appearances of the various symbols within the context c along 
the sequence z". That is, for all /3 G Z, m.{z"' , zZj., Zi) is the 1 2^ | -dimensional column vector whose /3-th 
component is 

m(z",zI^,z^)[/3] = \{t:k+l<t <n-k,zltl = zZlPzf}\, 

namely, the number of appearances of zZ\.fiz\ along the sequence z". 

The main result of [1] is the following theorem, pertaining to the semi-stochastic setting of an individual 
sequence x = (xi, X2, . . .) corrupted by a DMC that yields the stochastic noisy sequence Z = (Zi, Z2, . . .). 

Theorem 1 ([1, Theorem 1]) Take k = kn satisfying kn\2\'^^" = o(n/ log n). Then, for all x G the 
sequence of denoisers {X"^^^} defined in (16) satisfies: 



a) 



b) 



lim 

n— >oo L 



L^„.fc„(x'^,Z")-£>fe„(x",Z'^) 



E 



L^..fc„(x",Z")-L>fe„(x'*,Z") 



univ 



O 



a.s. 



' kn\Z\'^''" 

n 
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Theorem 1 was further shown in [1] to imply the universahty of the DUDE in the fully stochastic setting 
where the underlying sequence is emitted by a stationary source (and the goal is to attain the performance 
of the optimal distribution-dependent denoiser). 

Prom (16), it is apparent that the DUDE ends up employing a A;-th order sliding window denoiser 

(where the sliding window scheme the DUDE chooses depends on z"). Moreover, (9) implies that, at each 
time t, DUDE is merely employing the single-symbol denoiser i?//(m(z", z^Z^, -z^^i ), •) ^ 'S, which can be 
obtained by finding the Bayes response 6£(m(z", z^zl, ^l^i)) or, equivalently, the mapping in S given by 

argmin ^ i{zr,s), (17) 

rer(ct) 

where £{z,s) is the loss function defined in (6). By comparing (15) with (17), and from Theorem 1, we 
observe that working with the estimated loss £{zt,s) in lieu of the genie-aided A{xt-,s{zt-)) allows us to 
essentially achieve the genie-aided performance in (14). 

2- C Motivation 

Our motivation for this paper is based on the observation that the A;-th order sliding window denoisers ignore 
the time- varying nature of the underlying sequence x"^. That is, as discussed above, for time instances with 
the same contexts, the single-symbol denoiser employed along the associated subsequence is time-invariant. 
In other words, for each t, only the empirical distribution of the sequence {{xt,Zt) ■ t G T(ct)} matters, 
but its order of composition, i.e., its time- varying nature, is not considered. It is clear, however, that when 
the characteristics of the underlying clean sequence x'^ are changing, the (normalized) cumulative loss that 
is achieved by sliding window denoisers that can shift from one rule to another along the sequence may 
be strictly lower (better) than (14). We now devise and analyze our new scheme that achieves this more 
ambitious target performance. 

3 The Shifting Denoiser (S-DUDE) 

In this section, we derive our new class of denoisers and analyze their performance. In Subsection 3-A, 
we begin with the simplest case, competing with shifting symbol-by-symbol denoisers, or, in other words, 
shifting 0-th order denoisers. The argument is generalized to shifting fe-th order denoisers in Subsection 

3- B, and the framework and results include Subsection 3-A as a special case. We will use the notation »So, 
instead of S, for consistency in denoting the class of single-symbol denoisers. Throughout this section, we 
assume the semi-stochastic setting. 

3-A Switching between symbol-by-symbol (0-th order) denoisers 

Consider an n-tuple of single-symbol denoisers S = {si, • • • , s„} € Sq. Then, as mentioned in Section 2-B, 
for such S, we can define the associated n-block denoiser X"'^ as 

Xf{z^) = st{zt). (18) 

Note that in this case, the single-symbol denoiser applied at each time may depend on the time t (but not 
on as would be the case for a general denoiser). We also denote the estimated normalized cumulative 
loss as 

1 " 

Ls{z^) = -^i{zt,st), (19) 
^ t=i 

whose property is given in the following lemma, which parallels [2, Theorem 4]. 
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Lemma 1 Fix e > 0. For fixed S e Sq, and all G X"', 

2e2 



p(l^„.s(x", Z") - Ls(Z") > e) < exp ( - n-^) and (20) 



max 
2 



P(Ls(Z")-Lx„,s(x",Z") >€) < exp(-n-^), (21) 



"max 



^max ~ ^max- 



In words, the lemma shows that for every S E Sq, the estimated loss Ls(Z") is concentrated around the 
true loss Lj^n,s{x'^, Z'^) with high probability, as n becomes large, regardless of the underlying sequence 

x"'. 

Proof of Lemma 1: See Appendix 8- A. ■ 

Now, let the integer < m < [^J denote the maximum number of shifts allowed along the sequence. 
Then, define a set Sq ,^ C Sq as 

n 

= {S G 5o" : ^ Ms,-,^s,} < m}, (22) 

t=2 

namely, 5q ^ is the set of ra-tuples of single-symbol denoisers with at most m shifts from one mapping to 
another.^ Analogously to (14), for the class of n-block denoisers X"'^ with S G Sq^, we define 

Do,m{^'',zn ^ min L^„.s(x",z") 



1 " 



Se5oV n 

which is the minimum normalized cumulative loss that can be achieved for (x",z") by the sequence of n 
single-symbol denoisers that allow at most m shifts. Our goal in this section is to build a universal scheme 
that only has access to Z", but still essentially achieves Do^mix^ , Z^)- 

As hinted by the DUDE, we build our universal scheme by working with the estimated loss. That is, 
define 

S = S(z-) ^ arg min Ls(z"), (24) 

and our (0, rn)-Shifting Discrete Universal DEnoiser (S-DUDE), X"j|^^, is defined as X"'^. It is clear 
that, by definition, L-j^„ s(a;"', z") > 1)0,^(3^") -2^") for all and z", but we can also show that, with high 
probability, s(x"^, Z"-) does not exceed Do,m(2;") -^") by much, as stated in the following theorem. 

Theorem 2 Let X^^l^"* be defined as X"'^, where S is given in (24)- Then, for all e> and G X"', 

f(l^.,o,.(x",Z") - «.,„{.",Z-) > .) < 2exp (-„[^ - 2{ft(9 + ""^^""-" }] 

where h{x) = —x\nx — {1 — x)\n{l — x) for < X < 1, and N = \S\ = \Z\\'^\ . In particular, the right-hand 
side of the inequality is exponentially small, provided m = o{n) . 



Note that, when m = 0, Sq^ is the set of constant n-tuples consisting of the same single-symbol denoiser. 
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Remark: It is reasonable to expect this theorem to hold, given Lemma 1. That is, since, for fixed S G 5q ^, 
Ls{Z^) is concentrated on L-^„ s{x^,Z'^), it is plausible that S that achieves mingg^j^ ivs(^"') will have 
a loss L^„ s{x''\Z"-) close to m.msess^^L^„,s{x"', Z"-), i.e., Do,m(a;", ^"). 
Proof of Theorem 2: See Appendix 8-B. ■ 

3-B Switching between k-th order denoisers 

Now, we extend the result from Subsection 3-A to the case of shifting between k-ih order denoisers. The 
argument parallels that of Subsection 3-A. Let {sk,t}t=k+i arbitrary sequence of the k-th order 

denoiser mappings, i.e., Sk^t G Sk ioi k + 1 < t < n — k. Now, for given z", define an (n — 2fe)-tuple of (fc-th 
order denoiser induced) single-symbol denoisers 

S.(^") ^ {sk,t{ct, -mik+i e ^o-'', (25) 

where, to recall, Ct = (z*!^, z*^^'), and Sk,t{ct,-) is the single-symbol denoiser induced from Sk,t £ «5jfc and 
Cf. For brevity of notation, we will suppress the dependence on in 8^(2;"') and denote it as S^. Then, 
as in (18), we define the associated ra-block denoiser X"'^* as ^ 

±f''iz^) = sk,tict,zt). (26) 
In addition, extending (19), the estimated normalized cumulative loss is given as 

-. n—k 

^^"^'^^ = 5Z ^{zt,Sk,t{ct,-)). (27) 

t=k+l 

Then, we have the following lemma, which parallels Lemma 1. 

Lemma 2 Fix e > 0. For any fixed sequence {sk,t}t=k+i' ^ '^"^ 

Pr(L^„,s,(x^-^Z")-Ls,(Z")>e) < (fc + 1) exp |^ - j and (28) 

Pr(Ls,(Z")-L^„,s,(4;f,Z«)>.) < (^ + 1)«^p(-|i:T^)' (^9) 

where -i'max ^max ~l~ ^max- 



Remark: Note that when k = 0, this lemma coincides with Lemma 1. The proof of this lemma combines 
Lemma 1 and the de-interleaving argument in the proof of [1, Theorem 2]. Namely, we de-interleave into 
(fc -|- 1) subsequences consisting of symbols separated by blocks of k symbols, and exploit the conditional 
independence of symbols in each subsequence, given all symbols not in that subsequence, to use Lemma 1. 
Proof of Lemma 2: See Appendix 8-C. ■ 

Now, for an integer < m < [^^-^J and given z", let n(c) = |T(c)|, and m(c) = min{n(c),m} for 
c G Cfe. Then, analogously as in (22), we define 

Simian = {Skizn G 5o"-2^ : {Sk,r{c, ■)}renc) G for all c G C^}. (30) 



^Again, the value of («") ioi t < k and t > n — k can be defined as an arbitrary fixed symbol, since it will be 
inconsequential in subsequent development. 
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In words, SJ^^{z^) is the set of (n — 2A;)-tuples of {k-th order denoiser induced) single-symbol denoisers 
that allow at most m(c) shifts within the subsequence {t : t £ T{c)} for each context c € C^.^^ Again, for 
brevity, the dependence on in ^'^^^{z'^) is suppressed, and we write simply SJ^^. It is worth noting that 
(S^^ is a larger class than the class of A;-th order 'sliding window' denoisers that are allowed to shift at 
most m times. The reason is that in the shift within each subsequence associated with each context 

can occur at any time, regardless of the shifts in other subsequences, whereas in the latter class, the shifts 
in each subsequence occur together with other shifts in other subsequences. 

For integers A; > and n > 2k, we now define, for the class of n-block denoisers X"'^ with S G 5^^, 

D,,m{x'',zn ^ min L^„,s(4-^ z") 



n—k 



= Yl Hxt,sk,t{ct,zt)), (31) 



Se5,"^ n-2k ^^^^^ 



the minimum normalized cumulative loss of (x'^,z") that can be achieved by the sequence of fc-th order 
denoisers that allow at most m shifts within each context. Now, to build a legitimate (non genie-aided) 
universal scheme achieving (31) on the basis of only, we define 

Sfc,m = arg min Lsl^"), (32) 



and the (fc, m)-S-DUDE, X-'^^,^, is defined as X'*'^'=.'". Note that when m = 0, X"'^'=.'" coincides with the 
DUDE in [1]. The following theorem generalizes Theorem 2 to the case of general > 0. 

Theorem 3 Let X.'^Z be given by X.'^'^k.m^ where S^^^ is defined in (32). Then, for all e > and 

Pr(L^„..,^(x^;^, Z-) - Dk,mix\ Z^) > e) (33) 

\ univ / 



< 2{k + 1) exp (-in-2k). \ „ , - 2|Zp^ • \h(^) + (_!!1±}}}^}] ] , 

- ^ y V J I2{k + 1)LI^^ ' ' L Vn-2A;/ n-2k )\J 

where h{x) = — xlnx — (1 — a;)ln(l — x) for < a; < 1, and iV = |<S| = l^^ll"^!. 



(34) 



Remark: Note that when k = 0, this theorem coincides with Theorem 2. Similarly to the way Theo- 
rem 2 was plausible given Lemma 1, Theorem 3 can be expected given Lemma 2, since S^^^ achieves 

of Ls(Z") to L^„,s(x^;f , Z") for all S € 
Proof of Theorem 3: See Appendix 8-D. 



mms(zsj^^^Ls{Z'^), and we expect ^ (a;^_|_J^, Z") to be close to D^^^^^x^ , Z"^) from the concentration 



From Theorem 3, we now easily obtain one of the main results of the paper, which extends Theorem 
1 from the case m = to the case of general < m < [ "^^^' J . That is, the following theorem asserts 
that, for every underlying sequence x G X°° , our (A;, m)-S-DUDE performs essentially as well as the best 
shifting k-ik order denoiser that allows at most m shifts within each context, both in high probability and 
expectation sense, provided a growth condition on k and m is satisfied. 

Theorem 4 Suppose k = kn and m = rUn are such that the right-hand side of (34) is summable in n. 
Then, for all x G the sequence of denoisers {X^^^^"*} satisfies 

■^When m = 0, Si^q{z") becomes the set of n-block A;-th order 'sliding window' denoisers. 



10 



a) 



b) For any 5 > 0, 



lim 



a.s. 



(35) 



E 



= o 



hri. \ Z 



V n . 



1-5 



(36) 



Remark: It will be seen in Claim 1 below tliat the stipulation in tlie theorem implies limjj_>Qo 
0, which, when combined with (36), implies that the expected difference on the left hand side of (36) van- 
ishes with increasing n. That in itself, however, can easily be deduced from (35) and bounded convergence. 
The more significant value of (36) is in providing a rate of convergence result for the 'redundancy' in the 
S-DUDE's performance, as a function of both k and m. In particular, note that for any ij > 0, 0(n~^^'^~^^) 
is achievable provided kn = clogn and m„ = n^, for small enough positive constants c, ^. 

In what follows, we specify the maximal growth rates for k = kn and m = rrin under which the 
summability condition stipulated in Theorem 4 holds. 

Claim 1 a) Maximal growth rate for k: The summability condition in Theorem 4 is satisfied provided 
kn = c\ log n with ci < ^log \z\ ^^'^ grows at any sub-polynomial rate. On the other hand, the 
condition is not satisfied for kn = ci logn with any ci > 2\^\z\' ^I'en when m is fixed (not growing 
with n). 

b) Maximal growth rate for m: The summability condition in Theorem 4 is satisfied for any sub-linear 
growth rate ofrun, provided kn is taken to increase sufficiently slowly that kn\Z\^^" = o{{n/mnY~^) 
for some 5 > 0. On the other hand, the condition is not satisfied whenever rUn grows linearly with 
n, even when k is fixed. 

Proof of Claim 1: See Appendix 8-E. ■ 



Proof of Theorem 4- See Appendix 8-F. ■ 



3-C A "strong converse" 

In Claim 1, we have shown the necessity of m = o(n) for the condition required in Theorem 4 to hold. 
However, we can prove the necessity of m = o(n) in a much stronger sense, described in the following 
theorem. 

Theorem 5 Suppose that X = X , that A(x,x) > for all x,x with equality if and only if x = x, and that 
Il{x,z) > for all x,z. If m = @{n), then for any sequence of denoisers {X"}, there exists x°° G X°° 
such that 

lim sup £; [Lx„(x", Z") - Do,m(x", Z^)] > 0. (37) 

n— »oo 

Remark: The theorem establishes the fact that when m = o{n) does not hold, namely, when m = B(n), 
not only does the almost sure convergence in Theorem 4 not hold but, in fact, even the much weaker 
convergence in expectation would fail. Further, it shows that this would be the case for any sequence of 
denoisers, not necessarily the S-DUDE. Furthermore, (37) features Dq „i{x"', Z"'), pertaining to competition 
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with a genie that shifts among single-symbol denoisers so, a fortiori, it implies that for any fixed A; > or 
k that grows with n, 

lim sup E [L^„ (x", Z") - Z)fc,™(x", Z")] > (38) 

n— >oo 

also holds since, by definition, DQ^m{x^,z"') > Dk-mix^^z^) for all x",z"' and A; > 0. Therefore, the 
theorem asserts that for any sequence of denoisers to compete with Dfc^^(x",Z"), even in expectation 
sense, m = o{n) is necessary. Finally, we mention that the conditions stipulated in the statement of the 
theorem regarding the loss function and the channel can be considerably relaxed without compromising the 
validity of the theorem. These conditions are made to allow for the simple proof that we give in Appendix 
8-G. 



4 The Stochastic Setting 

In [1], the semi-stochastic setting result, [1, Theorem 1], was shown to imply the result for the stochastic 
setting as well. That is, when the underlying data form a stationary process, [1, Section VI] shows that 
the DUDE attains optimum distribution-dependent performance. Analogously, we can now use the results 
from the semi-stochastic setting of the previous section to generalize the results of [1, Section VI] and 
show that our S-DUDE attains optimum distribution-dependent performance when the underlying data 
form a piecewise stationary process. We first define the precise notion of the class of piecewise stationary 
processes in Subsection 4-A, and discuss the richness of this class in Subsection 4-B. Subsection 4-C gives 
the main result of this section: the stochastic setting optimality of the S-DUDE. 



4-A Definition of the class of processes 7^{m„} 

Let P-^^ , • ■ ■ 1 ^'x^^ be a finite collection of M probability distributions of stationary processes, with com- 
ponents taking the values in X. Let A be a process with components taking the values in {!,... ,M}. 
Then, a piecewise stationary process X is generated by shifting between the M processes in a way specified 
by the "switching process" A, as we now describe. 

First, denote r{A^) as the number of shifts that have occurred along the n-tuple A^, i.e., 

n-l 

Thus, there are r(yl") + 1 "blocks" in A", where each block is a tuple of constant values that are different 
from the values of adjacent blocks. Now, for each 1 <i < r{A'^) + 1, we define 



n a i = r(A") + 1 



as the last time instance of the i-th block in A". In addition, define ro(A") = 0. Clearly, r(A") and rj(A") 
depend on A^ and, thus, are random variables. However, for brevity, we suppress the dependence on A^ 
when there is no confusion, and write simply r and Tj, respectively. 

Using these definitions, and by denoting P^n as the n-th order marginal distribution of A, we define a 
piecewise stationary process X by characterizing its n-th order marginal distribution Px" as 



Pxn{X'' = x'') = ^PA-(a")P(X" = x"|^" 



r+l 



= ^PAn(a")n4"^'\<:_,+i), (39) 



i=l 
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for each n. The corresponding distribution of the process X is denoted as Px-^ In words, X is constructed 
by fohowing one of the M probabihty distributions in each block, switching from one to another depending 
on A. Furthermore, conditioned on the realization of A, each stationary block is independent of other 
blocks, even if the distribution of distinct blocks is the same. This property of conditional independence is 
reasonable for modeling many types of data arising in practice, since we can think of the M distributions 
as different 'modes'; if the process returns to the same mode, it is reasonable to model the new block as a 
new independent realization of that same distribution. In other words, the 'mode' may represent the kind 
of 'texture' in a certain region of the data, but two different regions with the same 'texture' should have 
independent realizations from the texture-generating source. Our notion of a piecewise stationary process 
almost coincides with that developed in [21]. The main difference is that we allow an arbitrary distribution 
for the process A. 

Now, we define P{mn} to be the class of all process distributions that can be constructed as in (39) 
for some M, some collection ■ ■ ■ ,P^^ of stationary processes, and some switching process A whose 
number of shifts satisfies 

r(^") < m„ a.s. Vn. (40) 

In words, a process X belongs to^ V^nin} if and only if it can be formed by switching between a finite 
collection of independent processes in which the number of switches by time n does not exceed m„. 



4-B Richness of V{mn} 

In this subsection, we examine how rich the class V{mn} is, in terms of the growth rate m„ and the 
existence of denoising schemes that are universal with respect to V{mn}- First, given any distribution on 
a noiseless ra-tuple, Px", we define 

D(Pxn,n)^ min £;Lx„(X",Z'^), (41) 

where I>„ is the class of all n-block denoisers. The expectation on the right-hand side of (41) assumes 
that X'^ is generated from Px^ and that is the output of the DMC, 11, whose input is X". Thus, 
]D)(Px",n) is the optimum denoising performance (in the sense of expected per-symbol loss) attainable 
when the source distribution Px^ is known. 

What happens when the source distribution is unknown? Theorem 3 of [1] established the fact that^ 



lim 



ELi,„ (X",Z") -B(Px",n) =0 for all stationary Px- (42) 

"^DUDE ^ ' ^ ' 



Note that our newly-defined class of processes, Vlnin}, is simply the class of all stationary processes if one 
takes the sequence m„ to be m„ = for all n. Thus, assuming m„ = 0, (42) is equivalent to 



lim 



£;L^„^^^(X", Z"") - B{Pxn,n) = for all Pk e V{mn}. (43) 



At the other extreme, when m„ = n, V{mn} consists of all possible (not necessarily stationary) processes. 
We can observe this equivalence by having M = processes each be a constant process at a different 
symbol in X, and creating any process by switching to the appropriate symbol. In this case, not only does 



^{Px"}n>i is readily verified to be a consistent family of distributions and, thus, by Kolmogorov's extension theorem, 
uniquely defines the distribution of the process X. 

®The phrase "the process X belongs to P{m„}" is shorthand for "the distribution of the process X, Px, belongs to 'P{m„}" . 

'^When Px is stationary, the limit lim„_»oo D(Pxn , 11) = D(Px,n) was shown to exist in [1]. Thus, (42) was equivalently 
stated as lim„_»oo EL<tn = D(Px, H) in fl, Theorem 31. 

-^DUDE \ > / L > J 
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(43) not hold for the DUDE, but clearly (43) cannot hold under any sequence of denoisers. In other words, 
V{mn} is far too rich to allow for the existence of schemes that are universal with respect to it. 

It is obvious then that Vlnin} is significantly richer than the family of stationary processes whenever 
rUn grows with n. It is of interest then to identify the maximal growth rate of rUn that allows for the 
existence of schemes that are universal with respect to Vlnin}, and to find such a universal scheme. In 
what follows, we offer a complete answer to these questions. Specifically, we show that if the growth rate 
of m„ allows for the existence of any scheme which is universal with respect to T{mn}, the S-DUDE is 
universal, too. 

4- C Universality of S-DUDE 

Here, we state our stochastic setting result, which establishes the universality of (A;, m)-S-DUDE with 
respect to the class 

Theorem 6 Let k = kn and Tfi — TTifi satisfy the yvowth vate condition stipulated in Theorem J^^ in addition 
to lim„_>oo kn = oo. Then, the sequence of denoisers {X^^^^™} defined in Section 3 satisfy 

= for all Pk e V{mn}. (44) 

Remark 1: Recall that, as noted in Claim 1, = o(n) together with appropriately slowly growing 
A: = fcri is sufficient to guarantee the growth rate condition stipulated in Theorem 4. Hence, by Theorem 6, 
m = o(n) and the sufficiently slowly growing k = kn suffices for (44) to hold. Therefore, Theorem 6 implies 
the existence of schemes that arc universal with respect to V{mn} whenever increases sublinearly in 
n. Since, as discussed in Subsection 4-B, no universal scheme exists for ■P{m„} when m„ is linear in n, we 
conclude that the sub-linearity of is the necessary and sufficient condition for a universal scheme to exist 
with respect to V{mn}- Moreover, Theorem 6 establishes the strong sense of optimality of the S-DUDE, 
as it shows that whenever V{mn} is universally "competable" , the S-DUDE does the job. This fact is 
somewhat analogous to the situation in [21], where the optimality of the universal lossless coding scheme 
presented therein for piecewisc stationary sources was established under the condition that m = o(n). 
Remark 2: A pointwise result 

lim \L^u,k,^ {X'\ Z") - ]D)(Px" , n)l = a.s. 

n— +0O univ 

for all -Px € 'Pi'iTT'n}, which is analogous to [1, Theorem 4], can also be derived. However, we omit such 
a result here since the details required for stating it rigorously would be convoluted, and its added value 
over the strong point-wise result we have already established in the semi-stochastic setting would be little. 
Proof of Theorem 6: See Appendix 8-H. ■ 

5 Algorithm and Complexity 

5- A An Efficient Implementation of S-DUDE 

In the preceding two sections, we gave strong asymptotic performance guarantees for the new class of 
schemes, the S-DUDE. However, the question regarding the practical implementation of (32), i.e., obtaining 

Sfe,m = arg nnn Lsiz"^), 

for fixed k, m and n remains and, at first glance, may seem to be a difficult combinatorial optimization 
problem. In this section, we devise an efficient two-pass algorithm, which yields (32) and performs denoising 



lim 

n— >oo 



EL 



(x",z")-D(Pxn,n) 
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with linear complexity in the sequence length n. A recursion similar to that in the first pass of the algorithm 
we present appears also in the study of tracking the best expert in on-line learning [15, 16]. 

Prom the definition of 5^^, (30), we can see that obtaining (32) is equivalent to obtaining the best 
combination of single-symbol denoisers with at most m(c) shifts that minimizes the cumulative estimated 
loss along {t : t £ T(c)}, for each c G C^. Thus, our problem breaks down to |Cfc| independent problems, 
each being a problem of competing with the best combination of single-symbol schemes allowing m switches. 

To describe an algorithm that implements this parallelization efficiently, we first define variables. For 
(A;, m)-S-DUDE, let I = m + 1,J = N + 1, where = |5| = \Z\\^\. Then, a matrix Mt G R^""-^ is 
defined for A;-|-l<t<n — 2k, where Mt{i,j) for 1 < i < I and 1 < j < J — 1 represents the minimum 
(un-normalized) cumulative estimated loss of the sequence of single-symbol denoisers along the time index 
{t : T < t,Cr = Ct}, allowing at most {i — 1) shifts between single-symbol denoisers and applying st = j- 
Moreover, Mt{i, J), for 1 < z < /, is the symbol-by-symbol denoiser that attains the minimum value of the 
i-th row of Mt, i.e., argmini<j<j_i Mt{i,j). A time pointer T G M"^, where D = |Cfc| = \Z\^^ , is defined 
to store the closest time index that has the same context as current time, during the first and second pass. 
That is, 

I N A / max{r : t < t,Cr = Cj}, when first pass 
* I min{T : T > t,Cr = Cj}, when second pass 

We also define r G and q G as variables for storing the pointer enabling our scheme to follow the 
best combination of single-symbol denoisers during the second pass. Thus, the total memory size required 
is 0{mNn + = 0{mn) (assuming that k satisfies the growth rate stipulated in the previous sections, 

which implies \Z\'^^ = o{n)). 

Our two-pass algorithm has ingredients from both the DUDE and from the forward-backward recursions 
of hidden Markov models [28] and, in fact, the algorithm becomes equivalent to DUDE when m = 0. The 
first pass of the algorithm runs forward from t = k + l to t = n — k, and updates the elements of Mj 
recursively. The recursions have a natural dynamic programming structure. For 2 < i < I, 1 < j < J — 1, 
Mt{i,j) is determined by 

Mt{i,j) = £(zt, j) + min{Mr(e,)(«,j),Mr(eo(^ " 1,^^(0,) (^ " 1,^))}, (46) 

that is, adding the current loss to the best cumulative loss up to T{ct) along {r : r < t, = c^}. When 
1 = 1, the second term in the minimum of (46) is not defined, and Mt{i,j) just becomes i{zt, j)+Mrp^^^^(i, j). 
The validity of (46) can be verified by observing that there are two possible cases in achieving Mt{i,j): 
either the {i — l)-th shift to the single-symbol denoiser j occurred before t, or it occurred at time t. 
We can see that the first term in the minimum of (46) corresponds to the former case; the second term 
corresponds to the latter. Obviously, the minimum of these two (where ties may be resolved arbitrarily), 
leads to the value of Mt{i,j) as in (46). After updating all Mj's during the first pass, the second pass runs 
backwards from t = n — k to t = k + l, and extracts 8^^^ from {Mt}^~^^^ by following the best shifting 
between single-symbol denoisers. The actual denoising (i.e., assembling the reconstruction sequence X^) 
is also performed in that pass. The pointers r{ct) and q{ct) are updated recursively, and they track the 
best shifting point and combination of single-symbol denoisers, respectively, for each of the subsequences 
associated with the various contexts. A succinct description of the algorithm is provided in Algorithm 1. 
The time complexity of the algorithm is readily seen to be 0{mn) as well. 
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Algorithm 1 The {k, m)-Shifting Discrete Denoising Algorithm 



^T(c,)(^'i) 



Require: Mt{i,j) G k + I < t < n - 2k,l < i < 1,1 < j < J, T e ,r G R^,qe R^, L G M 

Ensure: = {sk,t{ct, 

--k+i ("^^) denoised output {^tj^Lfe+i 

t(c) <^ (j) for all c G Cfe 
for t = k + 1 to n — 2k do 
if T(q) = (/) then 

Mt(i, j) ^ for 1 < i < /, 1 < j < J - 1 

Mt{i, J) <= argmini<j<j_i Mt{i,j) for 1 < i < I 

' MT(c,)ii,3) fori = l, 1< j< J-1 

mm{MTict)ii,j),MTict)ii-l,MT(ct)ii-^,J))} for 2 < i < /, l<j<J-l 
(z, i) ^ M*(^^) {i, j) + £{zt,j) for 1 < z < /, 1 < j < J - 1 
Mt(i, J) <;= argmini<j<j_i Mt{i,j) for 1 < i < / 
end if 

r(ct) <= t 

end for 

T(c) <^ (/> for all c G 

for t = n — 2k to k + 1 do 

if T(ct) = (j) then 
r(ct)^J, g(ct) ^Mt(r(ct), J) 

else 

L <= MTic,){r{ct),q{ct)) - i{zt,q{ct)) 
if L<Mt{r{ct),q{ct)) then 

r{ct) <= r{ct) - 1, q{ct) <= Mt{r{ct), J) 
end if 
end if 

Xt <^= Sk,t{Ct,Zt) 

end for 



5-B Extending the S-DUDE to Multi-Dimensional Data 

As noted, our algorithm is essentially separately employing the same algorithm to compete with the best 
shifting single-symbol denoisers, on each subsequence associated with each context. The overall algorithm 
is the result of parallelizing the operations of the schemes for the different subsequences, which allows for 
a more efficient implementation than if these schemes were to be run completely independently of one 
another. This characteristic of running the same algorithm in parallel along each subsequence enables us 
to extend S-DUDE to the case of multi-dimensional data: run the same algorithm along each subsequence 
associated with each (this time multi-dimensional) context. It should be noted, however, that the extension 
of the S-DUDE to the multidimensional case is not as straightforward as the extension of the DUDE was, 
since, whereas the DUDE's output is independent of the ordering of the data within each context, this 
ordering may be very significant in its effect on the output and, hence, the performance of S-DUDE. 
Therefore, the choice of a scheme for scanning the data and capturing its local spatial stationarity, e.g., 
Peano-Hilbert scanning [29], is an important ingredient in extending S-DUDE to the denoising of multi- 
dimensional data. Findings from the recent study on universal scanning reported in [30, 31] can be brought 
to bear on such an extension. 
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6 Experimentation 



In this section, we report some preliminary experimental results obtained by applying S-DUDE to several 
kinds of noise-corrupted data. 

6-A Image denoising 

In this subsection, we report some experimental results of denoising a binary image under the Hamming 
loss function. The first and most simplistic experiment is with the 400 x 400 black-and-white binary image 
shown in Figure 1. The first figure is the clean underlying image. The image is passed through a binary 
symmetric channel (BSC) with crossover probability (5 = 0.1, to obtain the noisy image (second image in 
Figure 1). Note that in this case, there are only four symbol-by-symbol denoisers, namely, S = {0,1, z, z}, 
representing always-say-0, always-say-1, say-what-you-see, and flip-what-you-see, respectively. The third 
image in Figure 1 is the DUDE output with k = 0, and the last image is the output of our S-DUDE with 
k = 0,171 = 1. The DUDE with A; = is competes with the best time-invariant symbol-by-symbol denoiser 




Figure 1: 400 x 400 binary images. 

which, in this case, is the say-what-you-see denoiser, since the empirical distribution of the clean image 
is (0.5,0.5) and 6 = 0.1. Thus, the DUDE output is the same as the noisy image; hence, no denoising is 
performed. However, it is clear that, for this image, the best compound action of the symbol-by-symbol 
denoisers is always-say-0 for the first half and then a shift to always-say-1 for the remainder. We can see 
that our (0, 1)-S-DUDE successfully captures this shift from the noisy observations, and results in perfect 
denoising with zero bit errors. 

Now, we move on to a more realistic example. The first image in Figure 2, a concatenation of a half- 
toned Einstein image (300 x 300) and scanned Shannon's 1948 paper (300 x 300), is the clean image. We 
pass the image through a binary symmetric channel (BSC) with crossover probability 6 = 0.1, to obtain 
the second noisy image, which we raster scan and employ the S-DUDE on the resulting one-dimensional 
sequence. Since the two concatenated images are of a very different nature, we expect our S-DUDE to 
perform better than the DUDE, because it is designed to adapt to the possibility of employing difi'erent 
schemes in difi'erent regions of the data. The plot shows the performance of our (k, m)-S-DUDE with 
various values of k and m. The horizontal axis refiects k, and the vertical axis represents the ratio of bit 
error per symbol (BER) to S = 0.1. Each curve represents the BER of schemes with different m = 0, 1, 2, 3. 
Note that m = corresponds to the DUDE. Wc can see that S-DUDE with m > mostly dominates the 
DUDE, with an additional BER reduction of ~ 11%, including when k = 6, the best k value for the DUDE. 
The bottom three figures show the denoised images with {k,m) = (4,0), (4,2), (6, 1), achieving BERs of 
5 X (0.744,0.6630,0.4991), respectively. Thus, in this example, (4,2)-S-DUDE achieves an additional BER 
reduction of 11% over the DUDE with k = A, and the overall best performance is achieved by (6, 1)-S- 
DUDE. Given the nature of the image, which is a concatenation of two completely different types of images, 
each reasonably uniform in texture, it is not surprising to find that the S-DUDE with m = 1 performs the 
best. 
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Figure 2: Clean and noisy images, the bit error rate plot for (k, m-)-S-DUDE, and three denoised outputs 
for {k,m) = (4, 0), (4, 2), (6, 1), respectively. 

6-B State estimation for a switching binary hidden Markov process 

Here, we give a stochastic setting experiment. A switching binary hidden Markov process in this example 
is defined as a binary symmetric Markov chain observed through a BSC, where the transition probabilities 
of the Markov chain switches over time. The goal of a denoiser here is to estimate the underlying Markov 
chain based on the noisy output. 

In our example, we construct a simple switching binary hidden Markov process of length n = 10^, in 
which the transition probability of the underlying binary symmetric Markov source switches from p = 0.01 
to p = 0.2 at the midpoint of the sequence, and the crossover probability of BSC is 5 = 0.1. Then, 
we estimate the state of the underlying Markov chain based on the BSC output. The goodness of the 
estimation is again measured by the Hamming loss, i.e., the fraction of errors made. Slightly better than 
the optimal Bayesian distribution-dependent performance for this case can be obtained by employing the 
forward-backward recursion scheme, incorporating the varying transition probabilities with the help of a 
genie that knows the exact location of the change in the process distribution. Figure 3 plots the BER of 
(k, m)-S-DUDE with various k and m, compared to the genie-aided Bayes optimal BER. The horizontal 
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axis represents k, and the two curves refer to m = (DUDE) and m = 1. The vertical axis is the ratio of 
BER toS = 0.1. 



Bit error rate plot for (k,m)-S-DUDE 
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Figure 3: BER for switching binary hidden Markov process {6 = 0.1, n = 10^). The switch of the underlying 
binary Markov chain occurs when t = 5 x 10^, from the transition probability p = 0.01 to p = 0.2. 

We can observe that the optimal Bayesian BER is (lower bounded by) 0.4865 x 6. The best performance 
of the DUDE was achieved when k = 6 with a BER of 0.5738 x 6, which is far above (18% more than) the 
optimal BER. It is clear that, despite the size of the data, the DUDE fails to converge to the optimum, as 
it is confined to be employing the same sliding-window scheme throughout the whole data. However, we 
can see that the (4, 1)-S-DUDE achieves a BER of .4979 x 6, which is within 2.3% of the optimal BER. 
This example shows that our S-DUDE is competent in attaining the optimum performance for a class 
richer than that of the stationary processes. Specifically, it attains the optimum performance for piecewise 
stationary processes, on which the DUDE generally fails. 

7 Conclusion and Some Future Directions 

Inspired by the DUDE algorithm, we have developed a generalization that accommodates switching be- 
tween sliding window rules. We have shown a strong semi-stochastic setting result for our new scheme 
in competing with shifting k-th order denoisers. This result implies a stochastic setting result as well, 
asserting that the S-DUDE asymptotically attains the optimal distribution-dependent performance for the 
case in which the underlying data is piecewise stationary. We also described an efficient low-complexity 
implementation of the algorithm, and presented some simple experiments that demonstrate the potential 
benefits of employing S-DUDE in practice. 

There are several future research directions related to this work. The S-DUDE can be thought of as a 
generalization of the DUDE, with the introduction of a new component captured by the non-negative integer 
parameter m. Many previous extensions of the DUDE, such as the settings of channel with memory [34], 
channel uncertainty [33], applications to channel decoding[37], discrete-input, continuous-output data[35], 
denoising of analog data[32], and decoding in the Wyner-Ziv problem[36], may stand to benefit from a 
revision that would incorporate the viewpoint of switching between time- invariant schemes. Particularly, 
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extending S-DUDE to the case where the the data are analog as in [32] wih be non-trivial and interesting 
from both a theoretical and a practical viewpoint. In addition, as mentioned in Section 5, an extension of 
the S-DUDE to the case of multi-dimensional data is not as straightforward as the extension of the DUDE 
was. Such an extension should prove interesting and practically important. Finally, it would be useful 
to devise guidelines, in the spirit of those in [38, 3], for the choice of k and m based on n and the noisy 
observation sequence z^. 
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8 Appendix 

8-A Proof of Lemma 1 

We first establish the fact that for all G Af", and for fixed S e Sq, 

is a {.Z„}-martingale. This is not hard to see by following: 
£;(n[Lx„.s(x",Z") -Ls(Z")]|Z"-i) 

n n 

= E[^A{xt,st{Zt))-^£iZt,st)\Z^'') 
t=i t=i 

= (n - l)[L^„_i,s(x"-\ - Zs(Z"-^)] + ^(A(x„, SniZn)) - iiZn, 

= (n-l)[L^„_i.s(x"-\Z"-i)-Ls(^"-^)], (47) 

where (47) follows from the fact that Zn is independent of Z^~^, and EA{xn, Sn{Zn)) = Ei(Zn, Sn)- 
Therefore, Ls{x^,Z^) — Ls{Z^) is a normalized sum of bounded martingale differences; therefore the 
inequalities (20) and (21) follow directly from the Hoeffding-Azuma inequality [14, Lemma A. 7]. ■ 

8-B Proof of Theorem 2 

Consider following chain of inequalities: 

p(Lx„.s(x",Z")-Z)o,™(x",Z") >e) 
= P( max {L^„.s(x^^") - L^„,s(x-, Z")} > e) 

^ E J'(i^xn.s(x",^")-Lx„,s(x",^")>e) (48) 

< P{Li^.,s{^'',Z-)-L^{Z-)>e/2)+ ^'(^s(^") " ^X".s(x^ ^") > 6/2) , (49) 
^ ^ ^ ^ 



(i) (ii) 
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where (48) follows from the union bound, and (49) follows from adding and subtracting Lg(Z"), and the 
union bound. For term (i) in (49), 



(i) < P( max {Lx„.s(x",Z")-Ls(Z")}>e/2) (50) 



where (50) follows from L^„_s (z", Z") - L^(Z") < maxses^"^ {L^^^six"", Z"-) - LsiZ"-)}, and (51) follows 
from the union bound and (20). Similarly, for term (ii) in (49), 

(ii) < Yl P{Ls{Z-)-L^„Ax'',Z^)>e/2) (52) 

max 

where (52) follows from L^{Z'^) < Ls{Z"') a.s., and (53) follows from (21). Therefore, continuing (49), we 
obtain 

,2 



(49) < 2^ J: exp(-n^) 



.[f("-).(.-.)fe.p(-„^) 

fc=0 ^ 

2(m + l)lnA' 



(54) 



< 2exp —n 



2L2 V n / n 



V n / n J / 



where (54) follows from \S^^J = J^Z^o {''^^) N {N - 1)'' , and (55) follows from \S^^J < iV™+i exp {nh{^)). 
Hence, the theorem is proved. ■ 

8-C Proof of Lemma 2 

We will prove (28) since the proof of (29) is essentially identical. As in [1] , define 

Id = {t:k + l<t<n-k,t = d mod(A: + l)}, 

whose cardinality is denoted = [(n — d — k)/{k + Then, by denoting Ct = {Z^Zl, -^t+i); we start 
the chain of inequalities. 



Pr(L^„.s,(x^;^,Z")-Ls,(Z")>6) 



^ f E E {H^r, SkACr, Zt)) - £{Zr, Sk,r{Cr, 0) } > " "^k)^ (56) 



d=0 \ reld 
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where (56) follows from the triangle inequality, (57) follows from the union bound, and {7^} is a set of 
nonnegative constants (to be specified later) satisfying ^^^7^ = 1- In the sequel, for simplicity, we will 
denote A(xt-, Sk^riC-,-, Zt)^ and £[Zt-, Sfc^^(C^, •)) in (48) as and i^, respectively. Now, the collection of 
random variables Z{d) is defined to be 

Z{d)^{Zt:l<t<n,t^Id}, 
and z{d) G Z"~"<^ denotes a particular realization of Z{d). Then, by conditioning, we have 



(57) < Yl Pr{Z{d)=z{d))Fv(Y.{^--^r}>{n-2k)jde 

d=0 z{d)eZ"-'^d \ TSXd 



Z{d) = z{d)\, (58) 



and let denote the conditional probability of (58). Now, conditioned on Z(d) = z{d), {Zt}t^j^ are all 
independent, and the summation in Pd beomes 



^ ^^A[xr,Sk,T{CT,ZT)) - i{Zr,Sk,T{CT,-))Y 



reld 



which is the sum of the absolute differences of the true and estimated losses of the symbol-by-symbol 
denoisers Sfc,r(cr) •) over r G Id- Thus, we can apply (20), and obtain 



Pd = Pr( V {Ar-£r} > 



{n - 2k)jde 
nd ■ - 



Z{d) = z{d) 



< exp - ^ ^ ^ . 59 



2^,2^2 
. nd 



Following [1], we choose 7^ = ^ and from the Cauchy-Schwartz inequality and 'Ylid'^^d = n — 2k, we 



-'3 

arrive at 



^<ik + l)Y,nd = {k + l)in-2k), 

^d. d=0 



and, hence, 

(fc + l)L2 



, 2(n-2k)e' . 

Pd < exp - -±——2—). (60) 



Therefore, plugging (60) into (58), we finally have 

/ N X f 2(n-2k)e^\ 

which proves the lemma. ■ 
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8-D Proof of Theorem 3 

The proof resembles that of Theorem 2. Consider 



(61) 



865; 



k,m 



ses, 



'^'^) -^s,J^") > i) -^x«,s(x^;f,^") > 0}(62) 



k,m 



n2|Cfe| 



865? 8g5, 



2(A; + 1) 



k=0 



exp 



(n - 2/c)e2 
2(fc + l)L2^, 



(63) 



(64) 



where (61)-(62) follow similarly as in (48)-(49); (63) follows from arguments similar to (50), (52), and 

Lemma 2 (which plays the role that Lemma 1 played there); and (64) follows from = [(X^^q'* ^ ' 

l)fcj|Cfe|_ ^^^^ ^ ^ 



m 



n-2k 



(65) 
(66) 



where (65) is based on the fact that exp(n/i(^)) is an increasing function in n, and (66) follows from 
m < [ ""2^^ J- Therefore, together with \Ck\ = we have 



(64) < 2{k + 1) cxp ( - (n - 2k) 
which proves the theorem. 



2|Z|2'=-U 



2ik + l)Ll^ 



m 



n-2k 



+ 



(m + l)lnAr- 
n-2k 



}])• <^^) 



8-E Proof of Claim 1 

2fclog \Z\ 

For part a), to show the necessity first, suppose ci > 2i5J]2|- Then, from \Z\ = n , we have 

2\Z\'^'' ■ {h{ ^^2k ') ^^n-2k ^ } ~ J7(ra 'osn (^)i-^)^ which will grow to infinity as n grows, even when 
m is fixed. Therefore, the right-hand side of (34) is not summable. On the other hand, k = cilogn 
with ci < is readily verified to suffice for the summability, provided that m = rrin grows at any 

sub-polynomial rate, i.e., grows more slowly than for any a > (e.g., C2 logn). 

For part b), to show the necessity, suppose m = 0(n). Then, ^( „!!^fc ) + ^^n-2k^ ~ ^^d, thus, 

for sufficiently smaU e, sTFFTTl^ l^l^'' ' i^iA) + ^"'V'-lT^ } < ^ ^ven for k fixed. Therefore, the 
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right-hand side of (34) is not summable. Hence, m = o{n) is necessary for the summabiUty. For sufficiency, 
suppose m = rUn is any rate, such that hm„_>oo ^ = 0. Then, 



2(/c + l)L2^^^ ' ' I Vn-2A;/ n - 2k 

1/ ofc|5r|2fc.nrfe^-^ 



n 



2k\Z\'"' ■0({^y-')\. (68) 



Thus, if k grows sufficiently slowly that klZl"^^ = o((^)^ then (68) becomes positive for sufficiently 
large n, and the right-hand side of (34) becomes summable. ■ 

8-F Proof of Theorem 4 

First, denote the random variable A^^ = L {x?T^, Z") - Dk,m{x'', Z«). Then, for part a), we have 

' univ 

L^n..,^(x", Z^) - Dk,m{x'', Z^) < ^ + Al^ a.s. 

univ n 

Since the maximal rate for k is ci logn as specified in Claim 1, lim„__>oo = 0. Furthermore, from the 

summability condition on k and m, Theorem 3, and the Borel-Cantelli lemma, we get lim^^oo m ~ ^ 
with probability 1, which proves part a). To prove part b), note that, for any e > 0, 



univ 



, 2fcAniax j_ rpf An \ 
^ + ^{^k,m) 

= + miJAlm < e)Pr{Alm <e) + E{AIJAI^ > e)Pr{Al^ > e) 

< ^^ + e + A^^.PriAl^>e) 
2kA 

< ^^^ + e + Amax- (right-hand side of (34)). (69) 

n 



Prom the proof of Claim 1, the condition of Theorem 4 requires k = kn and m = m„ to satisfy 

hm fc„|Z|2'="(!^)i-'5 = 0. 



n 



Therefore, if we set = Q{kn\2\'^^" ^) with sufficiently large constant then, from (68), we can see 

that the right-hand side of (34) will decay almost exponentially, which is much faster than 0(fc„|2p'^"(^)-^~'^) 

Hence, from (69), we conclude that E{A^j^^) = O fcn|-2p^"(^)^^^ , which results in part b). ■ 
8-G Proof of Theorem 5 

The fact that m = 0(n) implies the existence of a > 0, such that m > na for all sufficiently large n. Let 
X be the process formed by concatenating i.i.d. blocks of length [l/a], each block consisting of the same 
repeated symbol chosen uniformly from X. The first observation to note is that, for all n large enough 
that m > na, 

Do,m{X'\ Z'') = a.s. (70) 

This is because, by construction, is, with probability 1, piecewise constant with constancy sub-blocks 
of length, at least, [1/a]. Thus, a genie with access to X" can choose a sequence of symbol-by-symbol 
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schemes (in fact, ignoring the noisy sequence), with less than na (and, therefore, less than m) switches, 
that perfectly recover X" (and, therefore, by our assumption on the loss function, suffers zero loss). On 
the other hand, the assumptions on the loss function and the channel imply that, for the process X just 
constructed, 

limsuprnin£;L^„(X",Z") > 0, (71) 

n— »oo X" 

since even the Bayes-optimal scheme for this process incurs a positive loss, with a positive probability, on 
each \l/a] super-symbol. Thus, we get 

^jhmsup^; [L^„(X",Z») -Do,^(X^Z")|X'^] I (72) 

> limsupS[Lx„(X",^")- A,m(^",^")] (73) 

n^oo 

= limsupSLx„(X",Z") (74) 

n— >oo 

> lim sup niin EL-^„ (X^^ , ) 

n— >oo X" 

> 0, (75) 

where (73) follows from Fatou's lemma; (74) follows from (70); and (75) follows from (71). In particular, 
there must be one particular individual sequence x G for which the expression inside the curled brackets 
of (72) is positive, i.e., 

lim sup [Lxn(X",Z'^) - Do,miX'',Z^)\X^ = x'^] > 0, (76) 

n^oo 

which is equivalent to (37). ■ 



8-H Proof of Theorem 6 

First, by adding and subtracting the same terms, we obtain 

EL^r.,k,m (X", Z") - B(PX" , n) 

= EL^;Z(X",Z«)- min £;L^„,s(X^Z")+ min £;L^„.s(X^ Z") 



B(Pxn,n). (77) 



(i) 

We will consider term (i) and term (ii) separately. For term (i). 



(ii) 



(i) 



< 



< 



< 



EL^n,k,m{X'',Z'')- min £;L5j.„,s(X", Z**) 

n n 
2A;Ariiax 2A; 



n 

2kA m 



+ 



■E 



n 



E 



n 



univ 



(78) 
(79) 
(80) 



where (78) follows from upper bounding and omitting the losses for time instances t < k and t > n — k 
in the first and second terms of (i), respectively; (79) follows from exchanging the minimum with the 
expectation, and (80) follows from the definition (31) and 



n-2k 



< 1. 
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For term (ii), we bound the first term in (ii) as 



< 



min EL^^,s{X\Z^) 
2k{m + l)Ai, 



n 



H min E 



=1 j=n-i+k+i 



(81) 



by upper bounding the losses with Amax on the boundary of the shifting points. Now, let P v i An G M'"^' 

denote the | A" | -dimensional probabiUty vector whose x-th component is Pr{Xj = x\Z\,A^). Then, we can 
bound the second term in (81) by the following chain of inequalities: 



1 



min E 



i=l j=Ti-i+k+l 



n 



= -E 
n 



r+l n—k 

«=1 j=Ti-i+k+l 

r+l Ti—k 



min E 



K{X,,su{Z]tl)) 



EE E P^3^ = KXj,x)\Z'^_l = z\,A 



n 



= -E 
n 



n 



< -E 
n 



i=i j=Ti-i+k+i 2^ ^e2:2fc+i 

E E E pizp, = z'_,mu^{p 
E E ^[^A(Px,|zj+^AJl^' 

i=l j=Ti-i+k+l 

E E ^[^A(<ii,)i^" 

J=l j=ri_i+fc+l 
r+l r. 



(82) 
(83) 
(84) 

(85) 



E E 

2=1 j=Ti-l + l 



(86) 
(87) 



where (83) follows from the stationarity of the distribution in each block as well as the fact that the 
combination of the best k-th order sliding window denoiser for each block is in SJ^^^ and achieves the 
minimum in (82); (84) follows from conditioning; (85) follows from the definition (2); (86) follows from the 
stationarity of the distribution in each i-th block; and (87) follows from adding more nonnegative terms. 
For the second term in (ii), we first define 

n,{A^)^n{An-n-i{A'') 

as the length of the i-th block, for 1 < i < r(^") -|- 1. Obviously, ni{A^) also depends on A", and, thus, 
is a random variable, but we again suppress for brevity and denote it as n^. Then, similar to the first 
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term above, we obtain 



B{Pxn,U) = min EL^^iX"", Z"") 

r+l Ti 



— min E 



i=l j=rj_i+l 



= '-E 
n 



n 



= -E 
n 



n 



e[£ a(x,-,x,-(z-)) 

A(x,-,x(z;;_^+,)) 



min E 



> 



-E 



n 



r+l n 

Y. E 

i=l j=Ti-i+l 
r+l Ti 

> > min E 

1=1 :;=Ti_i+l 

1=1 j=Ti_l + l 

E E 

1 = 1 j = Ti_ 1+1 

r+l Ti 



(88) 



i=l j=ri_i+l 



(89) 
(90) 



where (88) follows from the conditional independence between different blocks, given vl"; (89) follows from 
the stationarity of the distribution in each block, and (90) follows from [1, Lemma 4(1)]. Therefore, from 
(81), (87), and (90), we obtain 



min ^L^„,s (^", Z"") - D(Pxn, H) 



^ 2fc(m + l)A^ax ^ 1^ 



n 



r+l 



2A;(m + 1)A„ 



n 



E E ^[f^A(p 

'- i=l j=Ti_i+l 
r+l 



,(^T,) 

Xq\Z°_ 



-^[C/a(P 



^"1 



i=l 



(91) 



Now, observe that, regardless of the sequence of numbers form a probability distribution, since 

S[=i ^ = 1 ^'^d ^ > for all i, with probability 1. Then, based on the fact that the average is less than 
the maximum, we obtain the further upper bound 



(91) < 



2k{m + l)An 



+ 



n 



The remaining argument to prove the theorem is to show that the upper bounds (80) and (92) converge 
to as n tends to infinity. First, from the given condition on = fc„ and m = m„, the maximal allowable 

growth rate for k is k = cilogn, which leads to lim„^oo = 0. In addition, the condition requires 

m = o{n), and A' to be sufficiently slow, such that k\Z\'^'' = o((^)^^*), which implies k = o(^). Therefore, 
lim„_>oo ^*-'^"'+-'-)Amax _ Q Pup^iigj-jjiore, from conditioning on X"", bounded convergence theorem, and part 
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b) of Theorem 4, we obtain lim„_oo E[L^n,k,m{X^~^ , Z") - Dk,m{X'^, ■^")] = 0. Thus, we have 

univ 



hm sup 

n— »oo 



< hm sup E 



max 



n^oo [ie{l,---,M} 



< E 
= 0, 



hm sup max 

n^oo i&{l,-,M} 



(93) 
(94) 



where (93) fohows from the reverse Fatou's lemma, and (94) fohows from [1, Lemma 4(2)] and M being 
finite. Since it is clear that liminf„^oo[-EL^«,fc,m(X", Z") - B(Px",n)] > by definition of B(Px«,n), 

univ 

the theorem is proved. ■ 



Remark: As in [1, Theorem 3], the convergence rate in (44) may depend on Px, and there is no vanishing 
upper bound on this rate that holds for all Px G ^{"^n}- However, we can glean some insight into the 
convergence rate from (i) and (ii): whereas the term (i) is uniformly upper bounded for all Px £ "Pfm^},^ 
the rate at which term (ii) vanishes depends on Px- In general, we observe that the slower the rate of 
increase of k = kn, the faster the convergence in (i), but the convergence in (ii) is slower. With respect 
to the rate of increase of m„, the slower it is, the faster the convergence in (i), but whether or not the 
convergence in (ii) is accelerated by a slower rate of increase of m„ may depend on the underlying process 
distribution Px- 
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